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DETAILED ACTION 
Arguments and Amendments 
1 . Applicant's amendments and arguments filed 7 August 2006, have been entered and 
made of record. 



Response to Amendments and Arguments 
2. Applicant has amended independent claims 24 and 33 to denote that "diagnostic 
information" is displayed. 

Applicant explains that diagnostic information can correspond to e.g. "scatter plot" pixel 
distributions described in the Specification at pages 1 1 and 12. It is apparent that neither 
Waxman, Watkins, nor Givens discloses or suggests the display of scatter plot information, 
which is an example of the claimed "diagnostic information." 

On pp. 10-11 of the Specification, another kind of diagnostic information is described - 
the separate display of image data from different sensors, such as visual (VIS) and short-wave 
infrared (SWIR) image data, before the data is fused together. Displaying the image data in this 
manner "allow[s] the operator to examine the output separately before the color fusion step." 

In a similar vein, Watkins discloses displaying image data from different sensors for 
"diagnostic" purposes. Figure 2 of Watkins shows the successive display of different 
combinations of fused image data and standard visible image data. Figure 2(a) shows the scene, 
as it would appear to the naked eye, using only the visible image data and none of the fused 
image data. Figure 2(b) shows an overlay of the fused image data at one-quarter intensity and the 
standard visible data at three-quarters intensity. Likewise, Figures 2(c) and 2(d) represent one- 
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half/one-half and three-quarters/qne-quarter combinations, respectively. See column 5/57 
through column 6/16. 

Such a successive display of the fused image data in combination with the standard 
visible image data is diagnostic in the sense that it M permit[s] the observer to see the true colors 
of the scene and at the same time observe the RGB enhanced stereo sensor fusion data." Column 
6/32-35. That is, it allows a viewer to more easily "diagnose" the scene visually - e.g., to identify 
targets that are only perceivable with the fused image data while observing the context around 
the target using the standard visible data. 

Claim Objections 

3. In light of Applicant's amendments, the previous claim objections have been withdrawn. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

5. This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
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invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 
6. Claims 24, 27-28 and 31-33, and 36-40 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent 5,555,324 by Waxman et al. ("Waxman") in view of U.S. Patent 
6,597,807 by Watkins et al. ("Watkins") and U.S. Patent 5,581,638 by Givens et al. ("Givens"). 

Regarding claims 24 and 33, Waxman discloses an image processing apparatus/method 
(figure 4) for processing imaging data in a plurality of spectral bands and fusing the data into a 
color image, comprising: 

two imaging sensors (cameras 3 1 0 and 312); 

at least two image-acquiring sensor areas located on said two imaging sensors (i.e. a first 
sensor area on camera 310 and a second sensor area on camera 312), wherein each said sensor is 
sensitive to a different spectral band than at least one other of said sensor areas (sensor area for 
camera 310 is sensitive to visible-near IR spectral band, and sensor area for camera 312 is 
sensitive to long-wave IR spectral band) and generates an image output representative of an 
acquired image in the spectral band to which the sensor area is sensitive (sensors for cameras 3 1 0 
and 312 produce visible/near IR and long-wave IR images, respectively); 

a frame grabber connected to said imaging sensors (A/D converters 328 digitize and 
transmit frames of image data to the processors); 

a general purpose computer (processor 330 is a general -purpose computer that performs 
various functions) connected to said imaging sensors for executing in real time: 
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a registration algorithm registering said image outputs (vision processor 392 contains 
processors 360 and 362 (see fig. 6) 5 which execute algorithms registering and generally 
preprocessing the image outputs); and 

a color fusion algorithm for combining said image outputs into a single image (fig. 6 
shows the execution of the algorithm for combining the VIS-NIR and LWIR image signals into a 
single RGB image). 

Waxman does not expressly teach that there are three image-acquiring sensor areas, as 
claimed. As shown in figure 4, Waxman utilizes a pair of image sensors, wherein each sensor is 
sensitive to a different spectral band. [The cameras share the same field of view and are focused 
along a common axis - col. 10/42-45.] 

Therefore, in Waxman's system, there are only two "image-acquiring sensor areas * * * 
sensitive to a different spectral band," as claimed. 

Also, Waxman discloses an operator interface as previously indicated, but Waxman does 
not disclose the display of "diagnostic information" on the interface, as claimed. 

Watkins discloses an image fusion system (figure 1) that is similar to Waxman's system. 
Like Waxman, Watkins utilizes cameras that are sensitive to more than one spectral band. The 
different spectral images captured from each sensor are fused into a single image according to 
RGB color codes. Then, the fused images of each stereo camera are displayed to e.g. the left and 
right eyes of a user - see col. 4/35-48. 

But whereas Waxman's cameras are sensitive in only two different spectral bands (VIS- 
NIR & LWIR), Watkins's cameras are each sensitive to three spectral bands (Near-IR, Mid-IR, 
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and Far-IR). Watkins's sensors thus contain "three image-acquiring sensor areas * * * sensitive to 
a different spectral band." 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Waxman by Watkins to achieve the claimed invention by employing three image- 
acquiring sensor areas sensitive to different spectral bands, as taught by Watkins. Watkins 
teaches that acquiring three different spectral images and then fusing those images together 
according to color for display to a user provides a representation of a viewed scene that 
capitalizes on the properties of the human visual system to offer enhanced target recognition 
capabilities. Specifically, capturing and displaying three different spectral images in the manner 
taught by Watkins takes advantage of a human's preattentive vision characteristics to allow 
objects in a scene to be identified more easily. See column 3/25-53 and column 4/48-67. 

Watkins discloses displaying image data from different sensors for "diagnostic" purposes. 
Figure 2 of Watkins shows the successive display of different combinations of fused image data 
and standard visible image data. Figure 2(a) shows the scene, as it would appear to the naked 
eye, using only the visible image data and none of the fused image data. Figure 2(b) shows an 
overlay of the fused image data at one-quarter intensity and the standard visible data at three- 
quarters intensity. Likewise, Figures 2(c) and 2(d) represent one-halfone-half and three- 
quarters/one-quarter combinations, respectively. See column 5/57 through column 6/16. 

Such a successive display of the fused image data in combination with the standard 
visible image data is diagnostic in the sense that it "permit[s] the observer to see the true colors 
of the scene and at the same time observe the RGB enhanced stereo sensor fusion data." Column 
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6/32-35. That is, it allows a viewer to more easily "diagnose" the scene visually - i.e., to identify 
targets that are only perceivable with the fused image data while observing the context around 
the target using the standard visible data. 

Based on the above teaching by Watkins, it would have been obvious to one of ordinary 
skill in the art at the time of the invention to modify Waxman by Watkins to display "diagnostic 
information" to an operator, as claimed. 

While Waxman teaches that the registration algorithm performed by preprocessors 360 
and 362 compensates for alignment, distortion errors, and the like (see column 13/37-40), 
Waxman does not expressly teach that the algorithm involves "scaling the image outputs," as 
claimed. 

Givens discloses a method for autonomous image registration for aligning two images 
having a common coverage area, such as those images obtained by Waxman in figure 6. In 
particular, Givens teaches that the process of registering two images having a common coverage 
area conventionally involves scaling down the images (see column 5, lines 36-45: the images are 
resampled at 1/8X magnification and then minified at a magnification of 1/32X magnification). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Waxman and Watkins by Givens to achieve the claimed invention by including a 
scaling operation of the images in the registration algorithm, as claimed, since Givens teaches 
that scaling common-coverage area images to be registered was a conventional image processing 
technique employed during an image registration process in order to reduce the images to a very 
low resolution (see column 5/36-45). The registration operations subsequently performed on the 
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images reduced by 32X would then be much less computationally intensive than if those 
operations were performed on the full resolution images. 

Regarding claims 27 and 36, the combination of Waxman and Watkins teaches the color 
fusion algorithm is simple color fusion whereby each sensor area image output is separately 
.assigned to a different color based on wavelength (Waxman, figure 6: colors are assigned on the 
basis of the wavelegnth being VIS-NIR or LWIR; and Watkins teaches that for each of the three 
different spectral images, RGB color codes are assigned based on the wavelengths thereof to 
create a single fused color image). 

Regarding claims 28 and 37, the combination of Waxman and Watkins teaches the color 
fusion algorithm is based on principle component color fusion whereby said sensor area outputs 
are fused into one image (Waxman, column 3, lines 28-43: the assigned RGB color codes are 
converted to the HVS color space to form a single, color- fused image). 

Regarding claims 31 and 38, Watkins teaches that each sensor comprises three sensors, 
each configured to map its image to an associated color channel (see figure 1), and a fusion 
algorithm is configured to combine the color channels into a color image (see figure 1 : the color 
channels for each stereo camera are combined into right and left composite video signals). 

Regarding claims 32 and 39, Watkins teaches the three sensors are sensitive to NIR, 
MWIR, and LWIR spectral bands (see figure 1). 

Regarding claim 40, the combination of Waxman and Watkins teaches the processing and 
fusing of said image occurs in real time (Waxman, column 1 1 , lines 1-4). 
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7. Claim 29 is rejected under 35 U.S.C. 1 03(a) as being unpatentable over U.S. Patent 
5,555,324 by Waxman et al. ("Waxman") in view of U.S. Patent 6,597,807 by Watkins et al. 
("Watkins") and U.S. Patent 5,581,638 by Givens et al. ("Givens"), as applied to claim 28, and 
further in view of U.S. Patent 4,533,938 by Hurst. 

Regarding claim 29, Waxman discloses performing a PCCF algorithm (i.e. converting 
RGB to HVS values) in order to perform "desirable color manipulations" within the HVS space 
(column 15, lines 5-21). Processor 520, figure 10, is capable of performing color transformations 
to produce desirable effects on the image to be displayed. 

Waxman does not disclose desaturating the HVS image, since Waxman's shunting 
algorithms produce images that are typically unsaturated (column 8, lines 48-51). However, 
should the final color image be saturated, it would have obvious to utilize processor 520 to 
desaturate the HVS image using any of the well-known desaturation techniques in order to 
enhance the image for display. 

For example, Hurst discloses an image processing system wherein a desaturation signal is 
added to the image signal when a hue value is modified (see Abstract). This desaturation signal 
is added in order to minimize the changes in saturation when the hue is modified. It would have 
been obvious to one of ordinary skill in the art at the time of the invention to modify Waxman by 
Hurst to desaturate the fused output image, as claimed, since Waxman teaches that a user to 
allowed to adjust the hue of the output image, and Hurst teaches that, as part of the logistics of 
adjusting the hue of an image signal, it is advantageous to desaturate the image so that changes in 
saturation attributable to the changing of the hue are minimized (see Abstract and column 1 , lines 
1 1-28 and column 2, lines 30-38). 
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Conclusion 

8. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 . 1 36(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Colin M. LaRose whose telephone number is (571) 272-7423. If 
attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Jingge 
Wu, can be reached on (571) 272-7429. The fax phone number for the organization where this 
application or proceeding is assigned is (571) 273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
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system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. Any inquiry 
of a general nature or relating to the status of this application or proceeding can also be directed 
to the TC 2600 Customer Service Office whose telephone number is (571) 272-2600. 



Colin M. LaRose 
Group Art Unit 2624 
15 October 2006 




